Repository navigation
Give an exported provider session a bounded shutdown grace and end unfinishable jobs - #272
Merged
Merged
Conversation
PollingController.Close now signals a private shutdown drain carried in the active invocation's context. A native launch holds that drain just before it exports auth, and no launch exports once the drain has begun. Only when a launch holds an export does the first Close wait, at most ten seconds (providerShutdownGrace) or its own earlier deadline, for the invocation to end before cancelling it as before. Every other phase is still cancelled at once, and a retried Close gets no second grace. During the grace the held launch closes its provider bridge, which closes the private provider stream and any in-flight request. The guest's existing client proxy stops its CLI on the failed exchange, and guest PID1 returns auth and the source audit in its ordinary result record. The host still proves whole-unit stop before any auth step and commits auth only through the durable stopped return. A failure after the drain reports cancelled, unless the guard latched a refusal or exhaustion, so the stopped job is requeued with its consumed budget. The engine starts no further phase once the drain began. A guest that does not return in time takes the existing stop-pending, fenced path. PollingController.Run's deferred Close budget becomes the grace plus its earlier ten seconds, inside GitReviewerService.Close's 30-second ceiling. The E2E close-provider branch where Close proves completion now expects the requeued job to rerun three phases and expire its captured budget, since the interrupted session stays charged, instead of a published review. Recovery of a lease whose stopped receipt is missing is unchanged and stays fenced pending the owner decision in #264. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GIT-STARTUP.md gains a shutdown grace section: what holds the drain, what Close waits for and for how long, why ten seconds fits the existing guest cleanup, native stop path and 30-second Close ceiling, and that stop proof and auth rules are unchanged. NATIVE-VM.md, POLLING.md and CONTROLLER.md describe the host revocation and controller behaviour. E2E-QUALIFICATION.md now names the Close-proves-completion branch of close-provider as the expected one, with the requeued job expiring its captured budget after three reruns, and keeps the fenced branch for a guest that does not return in time. Recovery of a lease whose stopped receipt is missing remains an open owner decision in #264 and stays fenced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…264) A job's captured session budget equals its schedule, one session per provider and round, so every charged session without a validated artifact leaves it one session short. After a shutdown grace the requeued job kept its interrupted session's charge and then reran three real sessions before expiring. sessionBudgetFits now states the rule from the job's configuration and checkpoint, as the engine plans phases, and such a job spends nothing: - the polling controller ends a retained queued or quota-waiting job that does not fit as incomplete_expired before restoring evidence, building a runtime or claiming it; - the claim ends it the same way before any lease, quota wait or session; - the engine starts no phase of a schedule that does not fit (ErrSessionBudgetExhausted); - ResumeUnavailable refuses to requeue it with ErrSessionBudgetExhausted, covering charged auth or provider failures and operator recovery. Its status names the cause: an incomplete_expired status reports the exhausted budget with its charged and resultless sessions, and an auth_failed or provider_unavailable status no longer promises a resume. The E2E close-provider branch where Close proves completion now expects the reopened job to end incomplete with no further launch and exactly one confirmed INCOMPLETE status matched to its remote COMMENT. Tests that relied on resuming a short budget now require the refusal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ome (#264) CONTROLLER.md states when a job's budget can no longer finish its schedule and every point that ends or refuses it, and the status wording that names the cause. GIT-STARTUP.md, POLLING.md and E2E-QUALIFICATION.md describe the requeued job after a shutdown grace ending incomplete with one status, and recovery's resume refusal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Collaborator
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This was referenced Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #264 (the bounded shutdown grace) and #176 (acceptance item 2). Recovery of a lease whose stopped receipt is missing is not implemented: it needs the owner decision recorded in #264 and stays fenced as before.
Shutdown grace
Closestarts the drain, no session exports.Closecancels at once, as before.providerShutdownGraceis 10 s or the caller's earlier deadline, only for the firstClose(a retry cancels at once), and never extends the session's own deadline. The guest's post-client cleanup is capped at 12 s and normally under 1 s; 10 s plus the host's 15 s stop path fits insideGitReviewerService.Close's 30 s ceiling.PollingController.Run's deferredClosebudget becomes the grace plus 10 s.ErrWorkerStopPending, recoverable through Recover a stopped provider attempt through an explicit operator action #251).cancelled, so the job requeues with its session charged; a session that completes during the grace keeps its artifact.Session budget rule
A requeued job keeps its charged session, and in the default schedule the budget equals the phases, so it can never finish; before this change it reran three more real sessions and expired.
sessionBudgetFits(engine.go) requires remaining sessions ≥ remaining phases. An unfinishable job is ended asincomplete_expiredbefore restore, runtime build, claim, lease or any session (Store.expireUnfinishable,claimJob);Engine.RunandResumeUnavailablerefuse it withErrSessionBudgetExhausted. Its status names the cause ("the captured session budget is exhausted…"), andauth_failed/provider_unavailablestatuses of an unfinishable job say it cannot be resumed. Behaviour change: after any charged failure in the default four-session schedule, a job ends instead of being resumable — the resume could never complete. A fresh owner request still creates a new job. Before-work refunds keep the budget whole, soquota_waitand refunded attempts still fit.Evidence
cancelled, refusals keep their class, a missing return stays fenced); budget tests (rule vs schedule, restarted controller ends the job with zero restores/builds/sessions and one INCOMPLETE status, unfinishablequota_waitended at claim,ResumeUnavailablerefusal). Fail-before: six grace behaviour tests and ten budget tests fail without the change.3dc6a4d(this branch on main with Record permanent capture refusals as coverage limits #270) on mini, from images, a source stub andreviewer.testbuilt from that commit, one VM at a time behind a ≥2.5 GB available-memory gate: exhaustion 12/12; hostile 19 pass + the containedcodex-metadataclient failure; every E2E case passes.close-providernow takes the grace branch (58 checks, 7.6 s): afterClosethe job is queued and unleased with 1 consumed session, the interrupted audit shows auth exported and returned, both auth stores are usable, and on reopen the job endsincomplete_expiredwith no further session and exactly one confirmed INCOMPLETE status (1 source / 1 native unit). The same results held at89c36a1on the previous base.go vet ./...andgo test -race ./...(reviewer package 267.174 s) pass.Coverage change
Because the guest now returns within the grace, the fenced branch is reached only when the guest hangs, so
close-provider-reopenandclose-provider-recoverno longer run on a host. #251's recovery keeps its unit and fault coverage; restoring an E2E path to the fenced branch is #271.Host-side behaviour only (service and
reviewer.test); every image and binary was rebuilt from the release commit for the measurement.🤖 Generated with Claude Code